Dynamic video overlays

ABSTRACT

A client device accesses a video input stream from an intermediate device for display. The client device analyzes the video input stream to determine that the video input stream matches a template corresponding to a screen portion. Based on the video input stream matching the template, a video output stream is generated and caused to be presented on a display. In some example embodiments, the analysis is performed while the client device is replacing video content received from a content source via the intermediate device. For example, commercials transmitted from a national content provider to a smart TV via a set-top box may be replaced with targeted commercials. During the replacement, menus generated by the set-top box may be detected and the replacement video altered by the smart TV to include the menus.

PRIORITY CLAIM

The application is a continuation of U.S. patent application Ser. No.15/381,497, filed Dec. 16, 2016, which claims priority to U.S.Provisional Patent Application No. 62/248,410, filed Dec. 16, 2015,entitled “Dynamic Video Overlays,” which are incorporated herein byreference in their entirety.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to the processingof video streams. Specifically, the present disclosure addresses systemsand methods to cause the presentation of an overlaid video.

BACKGROUND

Typically, people watch video content, such as television shows,advertisements, movies, video clips, and so on, via devices that receivea transmission from a content source. For example, a broadcaster (e.g.,HBO® or CNN®), a web server (e.g., YouTube®), a peer-to-peer source(e.g., another device), or another content source streams or otherwisetransmits video content to various devices capable of presenting thevideo content, such as televisions and associated set-top boxes,computing and/or mobile devices and associated media players orbrowsers, and so on.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings.

FIG. 1 is a block diagram depicting a network environment, in an exampleembodiment, for dynamically overlaying video.

FIG. 2 is a block diagram illustrating components of a videoidentification system and a query fingerprint generator, according tosome example embodiments.

FIG. 3 is a block diagram illustrating components of a client device anda replacement content server, according to some example embodiments.

FIG. 4 is a display diagram, in an example embodiment, illustrating adynamic video overlay.

FIG. 5 is a display diagram, in an example embodiment, illustrating adynamic video overlay.

FIG. 6 is a display diagram, in an example embodiment, illustrating adynamic video overlay.

FIG. 7 is a display diagram, in an example embodiment, illustrating adynamic video overlay.

FIG. 8 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay.

FIG. 9 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay.

FIG. 10 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay.

FIG. 11 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay.

FIG. 12 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay.

FIG. 13 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay.

FIG. 14 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay.

FIG. 15 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay.

FIG. 16 is a block diagram illustrating data structures, in exampleembodiments, for dynamically providing a video overlay.

FIG. 17 is a block diagram illustrating components of a machine,according to some example embodiments, able to read instructions from amachine-readable medium and perform any one or more of the methodologiesdiscussed herein.

DETAILED DESCRIPTION

Example methods and systems for dynamically providing a video overlayare described. In some example embodiments, a client device (e.g., asmart television (TV)) identifies a set-top box that presentsinformation overlaid on a video stream. Based on the identified set-topbox and video data received from the set-top box, a templatecorresponding to the presented information is selected. Based on thetemplate, a replacement video stream is presented along with theinformation, mimicking the manner in which the original video stream waspresented by the set-top box. In some example embodiments, thereplacement video stream is selected based on the original video stream.For example, the client device may generate a query fingerprint of aframe of the original video content. The client device queries adatabase of known reference fingerprints using the generated queryfingerprint, determines that the query fingerprint of the frame of videocontent matches a known reference fingerprint, and identifies the videocontent based on the match of fingerprints. Based on the identifiedvideo content, the replacement video stream is selected. For example, areplacement video stream showing a particular local commercial may beselected to replace an original video stream showing a particularnational commercial.

A fingerprint may be a single-frame fingerprint or a multi-framefingerprint. A multi-frame fingerprint is a collection of single-framefingerprints along with the relative timings of the frames. The term“fingerprint” is used to refer to both single-frame and multi-framefingerprints. The term “component” is used to refer to a single-framefingerprint within a multi-frame fingerprint, when confusion mightotherwise result.

In some example embodiments, the client device presents the replacementvideo stream at a particular position and size based on the identifiedtemplate. For example, the set-top box may be presenting the originalvideo stream as a reduced-size picture-in-picture. Accordingly, to mimicthe behavior of the set-top box, the replacement video stream ispresented at the same size and location as the original video stream.

In further example embodiments, the client device presents a particularportion of the replacement video stream based on the identifiedtemplate. For example, the set-top box may be presenting the originalvideo stream as a full-screen image, but covering the bottom of thestream with an informational bar. Accordingly, to mimic the behavior ofthe set-top box, only the top portion of the replacement video stream ispresented.

In additional example embodiments, the client device performs an alpha(transparency) blend of a particular portion of the replacement videostream based on the identified template. For example, the set-top boxmay be presenting a menu superimposed on the original video stream. Theclient device may extract the menu from the composite image andsuperimpose the menu on the replacement video stream using an alphavalue defined in the template.

In some example embodiments, the client device presents the replacementvideo stream at a particular speed based on the identified template. Forexample, the set-top box may be presenting the original video stream ina fast-forward mode. Accordingly, to mimic the behavior of the set-topbox, the replacement video stream is presented in a fast-forward mode.

In the following description, for purposes of explanation, numerousspecific details are set forth to provide a thorough understanding ofexample embodiments. It will be evident to one skilled in the art,however, that the present subject matter may be practiced without thesespecific details.

FIG. 1 is a network diagram illustrating a network environment 100suitable for dynamic video overlays, according to some exampleembodiments. The network environment 100 may include a watching station120 that receives video and other multimedia content from a contentsource 110, such as a broadcaster, web server, cable TV station, and soon. For example, the content source 110 may be a broadcaster, such as aTV station or TV network, which streams or transmits media over a TVchannel to the watching station 120, and/or a web service, such as awebsite, that streams or transmits media over a network 170 to thewatching station 120, among other things. The watching station 120includes a reference fingerprint generator 130 that generates referencefingerprints of video content received from the content source 110.

One or more set-top boxes 140 may also receive the video and othermultimedia content from the content source 110, such as via a broadcastchannel and/or over the network 170. The set-top box 140 may modify thereceived content before sending content to a client device 150. Theclient device 150 is any device capable of receiving and presenting astream of video and/or other multimedia content (e.g., a TV, a secondset-top box, a laptop or other personal computer (PC), a tablet or othermobile device, a digital video recorder (DVR), or a gaming device).

In some example embodiments, the set-top box 140 includes a tunerconfigured to receive an input stream of video content and generate anoutput stream of video content by processing the input stream. Theset-top box 140 may be a device equipped with tuners, decoders and soforth to access video content through a video content distributionnetwork, such as a terrestrial broadcast, cable, and/or satellitebroadcast network employed by multichannel video programmingdistributors. Additionally or alternatively, the set-top box 140 may bea device equipped with network adapters, decoders, and so forth toaccess video content through a wide area network (e.g., the internet)employed by devices for accessing internet video streaming services. Theoutput stream of video content may be comprised of multiple components.For example, the output stream of video content may include a firstcomponent in a first region of the output stream that corresponds to theinput stream and a second component in a second region of the outputstream that corresponds to video data generated by the set-top box 140.The second component may include a menu for the set-top box 140, aninformational message from the set-top box 140, or informationalmessages generated by other devices (e.g., phones or security alarms)and sent to the set-top box 140 for presentation. The set-top box 140outputs signals (e.g., digital or analog signals) usable by a display ofthe client device 150 to present the output video content to a userassociated with the client device 150. From the perspective of theclient device 150, the signals output by the set-top box 140 are a videoinput stream.

The client device 150 may also include a display or other user interfaceconfigured to display the processed stream of video content. The displaymay be a flat-panel screen, a plasma screen, a light emitting diode(LED) screen, a cathode ray tube (CRT), a liquid crystal display (LCD),a projector, and so on. In some example embodiments, the set-top box 140and the client device 150 are integrated into a single device.

The network 170 may be any network that enables communication betweendevices, such as a wired network, a wireless network (e.g., a mobilenetwork), and so on. The network 170 may include one or more portionsthat constitute a private network (e.g., a cable TV network or asatellite TV network), a public network (e.g., over-the-air broadcastchannels or the Internet), and so on.

In some example embodiments, a video identification system 190communicates with the watching station 120 and the client device 150over the network 170. The video identification system 190 may receive aquery fingerprint generated by a query fingerprint generator 160 of theclient device 150, such as a fingerprint of a frame or block of frameswithin the video content, and query an index of known referencefingerprints generated by the reference fingerprint generator 130 of thewatching station 120 in order to identify the video content by matchingthe query fingerprint with one or more reference fingerprints.

Upon identifying the video content, the video identification system 190may return an identifier for replacement content (e.g., alternativeprogramming, alternative commercials, and so on) associated with thevideo content to the client device 150. The replacement content may bestored in a replacement content server 180. Using the identifier, theclient device 150 may access the replacement content from thereplacement content server 180 and overlay the replacement content withvideo content received from the set-top box 140. For example, the clientdevice 150 may access and present replacement content from thereplacement content server 180, such as a replacement commercial for abroadcast channel.

Any of the machines, databases, or devices shown in FIG. 1 may beimplemented in a general-purpose computer modified (e.g., configured orprogrammed) by software to be a special-purpose computer to perform thefunctions described herein for that machine. For example, a computersystem able to implement any one or more of the methodologies describedherein is discussed below with respect to FIG. 16. As used herein, a“database” is a data storage resource and may store data structured as atext file, a table, a spreadsheet, a relational database, a documentstore, a key-value store, a triple store, or any suitable combinationthereof. Moreover, any two or more of the machines illustrated in FIG. 1may be combined into a single machine, and the functions describedherein for any single machine may be subdivided among multiple machines.

Furthermore, any of the modules, systems, and/or generators may belocated at any of the machines, databases, or devices shown in FIG. 1.For example, the video identification system 190 may include the queryfingerprint generator 160 and frames of video content from the clientdevice 150, and may generate the query fingerprints using the includedquery fingerprint generator 160, among other configurations.

FIG. 2 is a block diagram 200 illustrating components of a queryfingerprint generator 160, a reference fingerprint generator 130, and avideo identification system 190, according to some example embodiments.

The query fingerprint generator 160 of the client device 150 includes apatch selection module 210 and a value calculation module 220,configured to communicate with each other (e.g., via a bus, sharedmemory, or a switch). The video identification system 190 includes anindex module 230, a fingerprint match module 240, and an identificationmodule 250, all configured to communicate with each other (e.g., via abus, shared memory, or a switch). The reference fingerprint generator130 of the watching station 120 includes a patch selection module 260and a value calculation module 270, configured to communicate with eachother (e.g., via a bus, shared memory, or a switch). One or more of themodules described herein may be implemented using hardware (e.g., aprocessor of a machine, a field-programmable gate array (FPGA), or anapplication-specific integrated circuit (ASIC)) or a combination ofhardware and software (e.g., a processor configured by software).Moreover, any two or more of these modules may be combined into a singlemodule, and the functions described herein for a single module may besubdivided among multiple modules.

In some example embodiments, the query fingerprint generator 160 isconfigured and/or programmed to generate a query fingerprint of one ormore frames of video content captured at the client device 150. Forexample, the query fingerprint generator 160 may calculate values ofpatches, such as Haar-like features, regions, portions, and/or otheraspects of one or more frames within the video content. For example, apatch may be a portion of a frame having various different geometries,Haar-like features, and so on. In some example embodiments, some or allcaptured patches may each have a different scale and be at a differentlocation within a frame. By reporting fingerprints for patches of thereceived video content, the query fingerprint generator 160 maydetermine the content of each patch. Accordingly, if the video receivedcomprises multiple components from different sources (e.g., an inputvideo stream from a broadcaster and an overlay video stream generated bythe set-top box 140), the regions corresponding to the multiplecomponents can be identified.

The query fingerprint generator 160 (and the reference fingerprintgenerator 130) generates and/or creates fingerprints for identifyingvideo content from frames within the content. Typically, video contentreceived by the client device 150 will be in different formats andsample rates, and the query fingerprint generator 160 creates, for someor all of the frames of the video content, a query fingerprint for eachframe that is scale-independent and robust to different compressionartifacts. In some example embodiments, the query fingerprint generator160 may combine the query fingerprints of each of the frames to generatea query fingerprint of a block of frames (e.g., multiple frames) of thevideo content. The video identification system 190 may respond to videoidentification requests from millions of query fingerprint generators160.

The query fingerprint generator 160 may include a patch selection module210 configured and/or programmed to select multiple patches of the videocontent, such as patches associated with a displayed region of a frameor frames within the video content. Similarly, the reference fingerprintgenerator 130 may include a patch selection module 260 configured and/orprogrammed to select multiple patches of the video content, such aspatches associated with a displayed region of a frame or frames withinthe video content.

Patches may be created by dividing a frame into a grid (for example, a2×2 grid, a 4×3 grid, or a 4×4 grid). Patches may overlap. For example,20 patches may be used: four large patches corresponding to thequadrants of a frame and 16 small patches resulting from dividing theframe into a 4×4 grid. As another example, five patches may be used:four large patches corresponding to the quadrants of a frame and a fifthpatch of equal size located at the center of the frame. In someexamples, patches may be created by dividing a portion of a frame into agrid, such as a central portion of the frame that does not include theouter boundaries of the frame. Thus, the patches may cumulatively spanless than the entirety of the frame.

The query fingerprint generator 160 may also include a value calculationmodule 220 configured or programmed to calculate a value for each of theselected multiple patches using an integral image technique, such as atechnique that calculates the values using a summed area table or otherdata structure that generates a sum of values in a rectangular area of aregion. Similarly, the reference fingerprint generator 130 may include avalue calculation module 270 configured or programmed to calculate avalue for each of the selected multiple patches using an integral imagetechnique, such as a technique that calculates the values using a summedarea table or other data structure that generates a sum of values in arectangular area of a region.

For example, the patch selection module 210, 260 may select patches,such as Haar-like features that are commonly used in object detection,of regions of a frame or frames. The value calculation module 220, 270may utilize the Haar-like features to generate or calculate a same valuefor objects in a frame, such as objects in a visual image of a frame,regardless of the relative size of the object. For example, the valuecalculation module 220, 270 may generate values by approximatingGaussian filters and their derivatives using a box filter (e.g., anaverage of a region of the frame), wherein derivatives of the Gaussianfilters are created by finding the differences in the box filters.

The query fingerprint generator 160, via the value calculation module220, may generate a query fingerprint by calculating the values ofHaar-like features, or patches, at different scales and in differentlocations of displayed regions of the frames of video content. Thereference fingerprint generator 130, via the value calculation module270, may generate a reference fingerprint by calculating the values ofHaar-like features, or patches, at different scales and in differentlocations of displayed regions of the frames of video content.

In some examples, the patch selection module 210, 260 may select patchesthat include patches from a grid dividing a region of a frame or frames.The value calculation module 220, 270 may then calculate acharacteristic value for each of the patches, such as a summed value ofpixel values within each patch, a median value, etc. Moreover, the valuecalculation module 220, 270 may compute differences between thosecharacteristic values and/or linear combinations of those characteristicvalues. In some cases, the resulting differences and/or linearcombinations may then be used to determine a binary value (e.g., 0 or1), such as by comparison with a threshold. Thus, the fingerprintgenerator 160, 130, via the calculation module 220, 260, may generate afingerprint by calculating binary values that are based on differencesand/or linear combinations of characteristic values of multiple patchesof the frame or frames of video content.

The video identification system 190 includes the index module 230, thefingerprint match module 240, and the identification module 250, amongother modules, which are configured and/or programmed to match a queryfingerprint to a known reference fingerprint.

In some example embodiments, the index module 230 is configured and/orprogrammed to query a database of known reference fingerprints of videocontent captured at a reference device, such as the watching station120. For example, the index module 230 may query an index stored withina database of the watching station 120, an index stored within the videoidentification system 190, and so on.

For example, the index module 230 may be configured to query an index ofquantized patch values of the known reference fingerprints. The indexmodule 230 may query an index of 1-, 8-, 16-, and/or 24-bit numbers thatare associated with single frames of the reference fingerprints. Theindex module 230 may derive the numbers by quantizing one or more patchvalues of the reference fingerprints. For example, the index module 230may normalize fingerprint values by their absolute sum, by lognormalization, and so on.

The index module 230 may index some or all frames of the referencefingerprints using the best or better correlated values of large regionsof the frames. The index module 230 may create an index using highlycorrelated values of the full frame patches, because the featuresassociated with the best correlated values may represent the remainingpatches of a frame, among other things. The highly correlated values maybe pre-determined using a sample data set.

For example, when three regions of a frame are the best correlatedvalues, the index module 230 quantizes each of the three values to8-bits, such as by placing them in an evenly spaced histogram with amaximum and minimum limit, thereby creating a 24-bit number. The indexmodule 230 may then utilize a reference fingerprint index of 24-bitnumbers to quickly look up and/or identify frames. In some exampleembodiments, a fuzzy search is performed. The fuzzy search can findmatches that are inexact. For example, a fuzzy match for the three 8-bitvalues may be found when the matching values match exactly or are withinone of the corresponding value. In this example, there are 27permutations per index to parse when attempting to match fingerprints,with three matching values for each of the three bytes. Similarly, whenfour 8-bit values are used with a fuzzy match of the same degree (i.e.,with a maximum distance of 1 per value), there are 81 permutations perindex. Larger degrees and/or more values increase the number of possiblepermutations.

As another example, the index module 230 may quantize each of the threevalues to 1-bit, such as by comparing each value to a threshold, therebygenerating a 3-bit number. The index module 230 may then utilize areference fingerprint index of 3-bit numbers to quickly look up and/oridentify frames, because there are only 8 values per index to comparewhen attempting to match fingerprints.

In another example embodiment, 32 regions are used and each region isrepresented by a single bit. The 32-bit value may be hashed to generatea key. For example, if 256 hash values are used, the top 8 bits of the32-bit value may be used as the hash. To look up the fingerprint,potentially matching fingerprints with the same hash value are iteratedover and compared.

In some example embodiments, the fingerprint match module 240 isconfigured and/or programmed to determine that a query fingerprintmatches at least one known reference fingerprint. For example, thefingerprint match module 240 may determine that a query fingerprintmatches at least one known reference fingerprint by determining that asimilarity between the query fingerprint and at least one of the knownreference fingerprints satisfies a predetermined threshold associatedwith a Tanimoto distance measurement, a Manhattan distance measurement,and/or other distance measurements associated with matching images orother visual-based content.

For example, the fingerprint match module 240 may compare a queryfingerprint to one or more reference fingerprints using the Tanimoto orthe Manhattan distance measurements, and determine that the fingerprintsmatch when the comparison indicates that the distance measurementsatisfies a predetermined threshold (e.g., is within a certain distancethreshold). Of course, the fingerprint match module 240 may utilizeother matching techniques in order to determine whether a queryfingerprint matches a reference fingerprint, such as Euclidean, Cosine,KL-Divergence and/or Itakura distance measurement techniques, amongother distance measurement techniques.

In some example embodiments, the video identification system 190 mayutilize various different block sizes (e.g., number of frames, or framerate) of fingerprints when matching a query fingerprint to a referencefingerprint. For example, the reference fingerprint may be set at 5 fps,and occupy approximately 560 KB/h of runtime given an 8-bit value (32bytes/frame) for the fingerprint, and the query fingerprint, which mayinclude offset errors, may be set at 15 fps or higher. In this example,a query of the index may involve querying multiple reference frames(e.g., three frames for a reference fingerprint) of the referencefingerprint index. As another example, the reference fingerprint may beset at 15 fps while the query fingerprint is set at 5 fps. In thisexample, comparing the query fingerprint to the reference fingerprintmay involve comparing each of the query fingerprint frames to multiplereference frames of the reference fingerprint index.

Thus, in some example embodiments, the video identification system 190may optimize a match rate for matching a query fingerprint to areference fingerprint by modifying the block sizes of the fingerprints.For example, fingerprint match module 240 may match a query fingerprintto one second of reference video content, 0.5 second of reference videocontent, one minute of reference video content, and so on. In somecases, the precise time of the match may not be able to be determined.For example, if a video includes a still image, then any portion of thevideo showing the still image would match the entire length of the stillimage segment.

As discussed above, some patches may have different scales than otherpatches. Large-scale patches may be more important in identifyingmatches than low-scale patches. For example, using 1-bit values, a matchmay be identified based on the number of mismatched bits being below athreshold, with all bits being treated equally. As another example, amatch may be identified based on a number of mismatched low-scale bitsbeing below a first threshold and the number of mismatched large-scalebits being below a second threshold. The second threshold may be zero.

In some example embodiments, the identification module 250 is configuredand/or programmed to identify video content captured at a client device150 based on a determination that a query fingerprint matches at leastone reference fingerprint. For example, the identification module 250may identify the name or title of the video content, a location withinthe video content currently being presented by the client device 150, achannel or broadcaster providing the video content, and so on.

FIG. 3 is a block diagram 300 illustrating components of a client device150 and a replacement content server 180, according to some exampleembodiments. The client device 150 includes a set-top box identifiermodule 310, a video source identifier module 320, and a video overlaymodule 330, all configured to communicate with each other. Thereplacement content server 180 includes a video provision module 340 anda template provision module 350.

The set-top box identifier module 310 identifies the set-top box 140connected to the client device 150. For example, when the set-top box140 and the client device 150 are connected by a High-DefinitionMultimedia Interface (HDMI) cable, the Extended Display IdentificationData (EDID) of the set-top box 140 may serve to identify the set-top box140.

As another example, the set-top box identifier module 310 may present auser interface on the client device 150 through which a user canidentify the set-top box 140. This may be accomplished by having theuser manually identify the set-top box 140 by selecting a make and modelnumber from a drop-down list or entering the information into a textfield. The user-interactive identification may also be accomplished byinstructing the user to activate particular menu options of the set-topbox 140. The client device 150 accesses a database of known menus fromdifferent set-top boxes and compares them to the video data receivedafter the user activates the menu for the set-top box 140. When a matchis found, the set-top box 140 has been identified. In some exampleembodiments, multiple user actions on the set-top box 140 are needed touniquely identify the set-top box 140. For example, a first menu mayserve to cut the possible matches by 80%, a second menu may serve to cutthe possible matches by a further 80% (i.e., to 4% of the original setof possible matches), and a third menu may be needed to uniquelyidentify the set-top box 140.

The video source identifier module 320 identifies the source of videoprovided to the client device 150. In some example embodiments, thevideo source identifier module 320 is implemented by the queryfingerprint generator 160.

The video overlay module 330 combines video data received from thereplacement content server 180 with video data received from the set-topbox 140 based on the particular set-top box identified by the set-topbox identifier module 310. For example, a local commercial may beprovided by the video provision module 340 based on the identificationof a local video source by the video source identifier module 320. Thelocal commercial may be displayed as a picture-in-picture that overlaysa portion of the video data received from the set-top box 140. Theportion to be overlaid may be determined based on the identification ofthe set-top box 140 by the set-top box identifier module 310.

The video provision module 340 of the replacement content server 180provides replacement (or substitute) video content to the client device150 based on the identified video source. For example, the videoprovision module 340 may access a database that maps local commercialsfor a number of localities to national commercials for particularbroadcasters across the country. In this way, a particular nationalcommercial of a national broadcaster may be replaced by different localcommercials depending on the local broadcaster providing the signal tothe set-top box 140.

The template provision module 350 of the replacement content server 180provides templates to the client device 150 based on the identifiedset-top box. For example, set-top boxes by one manufacturer may have adifferent set of menu templates than set-top boxes by anothermanufacturer. Similarly, different models from the same manufacturer mayhave different sets of menu templates. Furthermore, different firmwarerevisions for the same model may have different sets of menu templates.Accordingly, a database that maps set-top box identifiers to sets oftemplates is maintained on the replacement content server 180 andtransmitted, in full or in part, to the client device 150 upon request.

Templates may be generated manually or programmatically. For manualgeneration, a person works with a client device and a set-top box and,by manipulation of the set-top box, causes presentation of each of thedifferent menus that can be generated by the set-top box. Throughanalysis of the presentation of the menus on a screen of the clientdevice, the person generates a template for each distinct menu format.The templates are stored in the database of the replacement contentserver 180 for later access.

For programmatic generation of the templates, one or more client devices(e.g., client device 150) and corresponding set-top boxes (e.g., set-topbox 140) are monitored while in use (e.g., through reporting of statusby the client devices to the replacement content server 180). In someexample embodiments, the client device or the replacement content server180 determines when the video received from the set-top box contains aportion of known video data (e.g., a recognized cable program) and aportion of unknown video data using fingerprinting of regions (e.g.,quadrants or sixteenths of the screen area). The unknown video data isassumed to be additional video content supplied by the set-top box.Accordingly, a pixel-level analysis of the received video data isperformed by the replacement content server 180 to determine the regionoccupied by the known video data, the region occupied by the unknownvideo data, and the region occupied by a blend of known and unknownvideo data (e.g., through alpha blending). From the regions, a templateis generated and, if not already stored in the database, stored in thedatabase of the replacement content server 180 for later access.

In some example embodiments, programmatic generation of the templatesinvolves monitoring multiple frames of video data received by the clientdevice from the set-top box. By comparing the sequential frames of videodata, regions of the video data that are changing rapidly can bedistinguished from regions that are static or undergo relatively fewchanges. From this analysis, it can be inferred that the rapidlychanging regions are displaying video programming and the static regionsare displaying information provided by the set-top box. Likewise, aregion that is rapidly changing but has an average color substantiallydifferent from that of the remainder of the video data may indicate thatblending of dynamic video with a static overlay is occurring.Accordingly, a blending region may also be identified through themulti-frame analysis. From the regions, a template is generated and, ifnot already stored in the database, stored in the database of thereplacement content server 180 for later access.

FIG. 4 is a display diagram 400, in an example embodiment, illustratinga dynamic video overlay comprising a region 410 and a region 420 of adisplay area of the client device 150. Video received from the set-topbox 140 is passed through for display in the region 410. Replacementvideo data is presented in the region 420. As can be seen in the exampleof FIG. 4, the region 420 is a rectangular region that is smaller thanthe display area of the client device 150. The region 420 also has noedges in common with the display area. In other example embodiments, theregion 420 has other shapes, sizes, and positions. In some exampleembodiments, a menu for the set-top box occupies the region 410 and theoriginal video data received from the content source 110 is scaled(e.g., shrunk), repositioned by the set-top box 140, and sent from theset-top box 140 to the client device 150 in the region 420. Accordingly,when the replacement video data is presented in the region 420, thevideo data received from the content source 110 is replaced while themenu shown in the region 410 remains visible to the user.

FIG. 5 is a display diagram 500, in an example embodiment, illustratinga dynamic video overlay comprising a region 510 and a region 520 of adisplay area of the client device 150. Video received from the set-topbox 140 is passed through for display in the region 510. The replacementvideo data received from the replacement content server 180 is croppedto, and presented in, the region 520. In some example embodiments, amenu for the set-top box 140 occupies the region 510 and a portion ofthe original video data received from the content source 110 is croppedby the set-top box 140 to the region 520 and sent from the set-top box140 to the client device 150 in the region 520. Accordingly, when theportion of the replacement video data is presented in the region 520,the video data received from the content source 110 is replaced whilethe menu shown in the region 510 remains visible to the user. As can beseen in FIG. 5, the top edge of the region 520 is identical to the topedge of the display area, and the left and right edges of the region 520are shared with the left and right edges of the display area, but theregion 520 does not occupy the full height of the display.

FIG. 6 is a display diagram 600, in an example embodiment, illustratinga dynamic video overlay comprising a region 610 and a region 620 of adisplay area of the client device 150. Video received from the set-topbox 140 is passed through for display in the region 610. The replacementvideo data received from the replacement content server 180, less theportion corresponding to the region 610, is presented in the region 620.In some example embodiments, a menu for the set-top box occupies theregion 610 and a portion of the original video data received from thecontent source 110 corresponding to the region 620 is sent from theset-top box 140 to the client device 150 in the region 620. Accordingly,when the portion of the replacement video data is presented in theregion 620, the video data received from the content source 110 isreplaced while the menu shown in the region 610 remains visible to theuser.

FIG. 7 is a display diagram 700, in an example embodiment, illustratinga dynamic video overlay comprising a region 710, a region 720, and aregion 730 of a display area of the client device 150. Video receivedfrom the set-top box 140 is passed through for display in the region710. The replacement video data received from the replacement contentserver 180, less the portion corresponding to the region 710, ispresented in the region 720. Video received from the set-top box 140 isblended with the replacement video data and presented in the region 730.For example, a gradient may be applied such that the portion of theregion 730 bordering the region 710 is entirely (or almost entirely)made up of video received from the set-top box 140, and the portion ofthe region 730 bordering the region 720 is entirely (or almost entirely)made up of the replacement video.

In some example embodiments, further image processing is applied in theregion 730 to reverse the application of image processing by the set-topbox 140. For example, the set-top box 140 may have blended informationprovided by the set-top box 140 with the input video stream. Thus, thisblended information may be considered to be a third video component,distinct from the first video component corresponding to the input videoto the set-top box 140 and the second video component corresponding tothe overlay information provided by the set-top box 140. To eliminatethe input video stream entirely from the output of the video overlaymodule 330, further filtering is applied to the region of the thirdvideo component (e.g., the region 730).

In some example embodiments, the filtering applied depends on thebrightness. Portions of the region 730 that exceed a predeterminedbrightness value are determined to be in the foreground and to containinformation that should be passed on to the user. Portions of the region730 that fall below the predetermined brightness value are determined tobe in the background, and thus any information contained should befiltered out. In other example embodiments, to filter out the backgroundinformation, the color of each background pixel is changed to theaverage color of all background pixels. For example, a light bluebackground superimposed over a video of a football game will, onceaveraged, maintain a light blue tint and not convey any informationabout the football game. In other example embodiments, the backgroundcolor is stored in the template. Accordingly, the color of theforeground pixels is maintained, and the remaining pixels are replacedwith the background color.

In further example embodiments, the image processing reverses the imageprocessing performed by the set-top box 140 using the input videostream. For example, by identifying the input video stream (e.g.,through fingerprinting), the original input video stream can be accessed(e.g., from the video identification system 190). By applying thereverse of the modification applied by the set-top box 140, theinformation overlaid by the set-top box 140 is recovered. For example,if the set-top box 140 generated the output video by blending theoverlay information at an 80% weight with the input video information ata 20% weight, then the client device 150 can reverse the process bysubtracting a 20% weight of the input video and multiplying theremainder by 1.25. The recovered overlay can then be applied to thereplacement video content by applying the same blend as was previouslyapplied by the set-top box 140. In this example, that would meangenerating pixels for display by taking an 80% weight of the overlaywith a 20% weight of the replacement video content.

FIG. 8 is a flowchart illustrating a method 800, in some exampleembodiments, for dynamically providing a video overlay. The method 800includes operations 810, 820, 830, 840, and 850. By way of example andnot limitation, the operations 810-850 are described as being performedby the components 110-190 of FIG. 1, the modules of 210-250 of FIG. 2,and the modules 310-350 of FIG. 3.

In operation 810, the set-top box identifier module 310 identifies aset-top box (e.g., the set-top box 140) providing video data to a clientdevice (e.g., the client device 150). For example, when the set-top box140 and the client device 150 are connected by an HDMI cable, the EDIDof the set-top box 140 may serve to identify the set-top box 140. Asanother example, the set-top box identifier module 310 may present auser interface on the client device 150 through which a user canidentify the set-top box 140 (e.g., by selecting from a list ofoptions). As a third example, the set-top box identifier module 310 maysend its internet protocol (IP) address to the replacement contentserver 180 (or another server) for lookup. In this example, thereceiving server maintains a database that maps IP addresses to cablecompany service areas and maps cable companies to their preferredset-top boxes. Accordingly, the IP address of the client device 150 maybe used to identify the type of device most likely to be the set-top box140.

Once the set-top box 140 is identified by the set-top box identifiermodule 310, an identifier for the set-top box 140 is stored in theclient device 150 for later access. For example, the user may beprompted on a first use of the client device 150 to identify the set-topbox 140, and not prompted on future uses since the information can beaccessed from storage.

The video source identifier module 320 identifies a portion of thereceived video data (operation 820). For example, the received videodata may include a broadcast TV signal overlaid with a menu generated bythe set-top box 140. Using the patch selection module 210 and the valuecalculation module 220 to generate one or more fingerprints for thereceived video data, the client device 150 sends the fingerprints to thevideo identification system 190. In response, the client device 150receives an identifier corresponding to the broadcast TV signal, whichserves to identify the portion of the received video data containing thebroadcast TV signal.

Alternatively, operation 820 may be performed by analyzing the videodata for specific patterns without fingerprinting. For example, aparticular video overlay provided by the set-top box 140 or the clientdevice 150 may include a brand name in a particular location on thescreen. Accordingly, that portion of the screen can be compared againsta captured image of the brand name to determine when the particularvideo overlay is present. As another example, a particular video overlaymay include an icon in a particular location on the screen or thenotification. To illustrate, a telephone icon may be shown in anotification regarding an incoming telephone call. Accordingly, variousportions of the screen are compared against a stored image of thetelephone icon to determine when the notification is present. Thepseudocode below is representative of some example embodiments:

templateID templateMatch(videoInStream, deviceID) {  // iterate overavailable templates for the intermediate device  // until a match isfound for the input stream  // return the templateID of the matchingstream, or 0 if none.  matchFound = 0;  do {   currentTemplate =getNextTemplate(deviceID);   if (singleTemplateMatch(videoInStream,currentTemplate))    matchFound = currentTemplate;  } until (matchFound!= 0)  return matchFound; } Boolean singleTemplateMatch(videoInStream,template) {  // compare the template to the input stream  // return TRUEif the template matches, FALSE otherwise  region = template.region; count = 0;  for (x = region.x; x < region.width; x++)   for (y =region.y; y < region.height; y++)    if((videoInStream.currentFrame(x,y) − template(x,y)) ** 2) <= PIXEL_MATCHDIFFERENCE)     count++;   // for exact pixel matching,PIXEL_MATCH_DIFFERENCE = 0  maxCount = x*y;  threshold =PERCENT_THRESHOLD * maxCount;  if (count >= threshold)   return TRUE; else   return FALSE; }

The example implementation of singleTemplateMatch( ), above, uses thesquare of the difference between corresponding pixels as a distancemeasure. Other example embodiments use the absolute value of thedifference, the normalized square of the difference, the correlationcoefficient, the normalized correlation coefficient, the crosscorrelation, the normalized cross correlation, or any suitablecombination thereof.

In operation 830, the client device 150 or the replacement contentserver 180 determines replacement video based on the identified videodata and/or profile information associated with the client device 150.For example, a PG-rated substitute version of R-rated content may beselected based on the identification of the particular R-rated content.As another example, a local commercial may be selected to replace aparticular national commercial.

The video overlay module 330, in operation 840, determines an overlaymask based on the identified set-top box and the video data. In someexample embodiments, the video data is compared to a set of known videoframes to determine if a particular menu or other informational elementis being displayed by the set-top box 140. For example, an informationalbar may be presented at the top of the screen by the set-top box 140, inwhich certain pixels are black and in which a certain area has anaverage pixel color within the range of (110,110,110)-(160,160,160),expressed as red-green-blue (RGB) color tuples. Accordingly, when oneframe, or a set of sequential frames, of the received video data has thecharacteristics of the informational bar, the video overlay module 330identifies a corresponding overlay mask, in which a pass-through portioncorresponds to the area occupied by the bar and a replacement portioncorresponds to the area occupied by the original video content receivedby the set-top box 140.

Templates, such as those corresponding to the examples presented inFIGS. 4-7, may indicate the region or zone of an overlay in severaldifferent ways. For example, coordinates of the boundaries of theoverlay zone may be provided by the template. Alternatively, a bitmap ofthe entire display may indicate the overlay zone (e.g., with a value of1 indicating that the corresponding pixel is part of the overlay and avalue of 0 indicating that the corresponding pixel is not part of theoverlay).

In the above example, the mask is selected to allow the user to see boththe overlay information generated by the set-top box and the replacementvideo content. In some example embodiments, the mask is selected tosuppress the overlay information generated by the set-top box. Forexample, optical character recognition (OCR) may be performed on theoverlay information to identify any text contained therein. Based on thetext, a database lookup may be performed to determine if the overlayshould be passed through or suppressed. When the overlay is to besuppressed, a mask is chosen that presents the replacement video contentin the region of the overlay. The two approaches may also be mixed,allowing some overlay content to be presented to the user and otheroverlay content to be suppressed. For example, if an overlay ispresenting text information associated with the input video content,such as a description of a product being displayed, the text informationis not likely to be relevant to the replacement video content.Accordingly, suppressing the text information improves the experience ofthe user in viewing the replacement video content.

In operation 850, the video overlay module 330 causes the display of aportion of the video data and a portion of the replacement video, usingthe mask. Continuing with the example above, the video data containingthe informational bar, corresponding to the pass-through portion of themask, is displayed along with a portion of the replacement videocorresponding to the replacement portion of the mask. In some exampleembodiments, the display of the portion of the replacement videocomprises repositioning or resizing the replacement video. For example,if the mask of FIG. 4 is used, the replacement video is reduced in sizeand repositioned to occupy the region 420.

In some example embodiments, the mask changes over time. For example,the overlaid information presented by the set-top box 140 may beanimated and change position over time. Accordingly, the location of theregions in which the received video data and the replacement video dataare presented may change over time. In some examples, changing the maskand/or size/location of replacement video over time may be carried outby repeating operations 840 and 850. For instance, operations 840 and850 may be repeated in an iterative fashion until the video overlaymodule 330 no longer determines that an overlay mask should be used(e.g., when the incoming video no longer matches a template having acorresponding mask). In some examples, changing the mask and/orsize/location of replacement video over time may be carried out based ona predetermined timed routine that is associated with a particular maskand/or template (e.g., an animation routine in which aspects of the maskchange over time in a predictable manner).

The method 800 may also be performed, with slight modifications, tocompensate for video elements provided by the client device 150. Thevideo elements provided by the client device 150 may be detected byanalysis of the video data, in the same manner as those provided by theset-top box 140. Alternatively, the client device 150 may report userinteractions to the video overlay module 330. For example, when theclient device 150 receives a user command to display a menu, thatinformation may be provided to the video overlay module 330. Based onthe particular menu being displayed, the video overlay module 330identifies the mask to apply.

FIG. 9 is a flowchart illustrating a method 900, in some exampleembodiments, for dynamically providing a video overlay. The method 900includes operations 810-850 and 910-950. The operations 810-850 aredescribed above, with respect to FIG. 8. By way of example and notlimitation, the operations 910-950 are described as being performed bythe components 110-190 of FIG. 1, the modules of 210-250 of FIG. 2, andthe modules 310-350 of FIG. 3.

In operation 910, the client device 150 detects a change in the videodata supplied by the set-top box 140 to second video data. For example,the user of the set-top box 140 may have changed the channel beingdisplayed.

In response to the detection of the change, the video overlay module 330ceases application of the mask and allows the entirety of the receivedvideo from the set-top box 140 to be displayed (operation 920).Accordingly, the user of the set-top box 140 will see the effect of thechanged channel.

In operations 930 and 940, the client device 150 detects a second changein the video data supplied by the set-top box 140. The second changereturns the received video data to the prior state. For example, theuser of the set-top box 140 may have returned to the original videochannel being displayed. The detection may be accomplished by receivinga set of expected fingerprints from the video identification system 190and comparing the expected fingerprints with the fingerprints generatedby the query fingerprint generator 160. Accordingly, when the userchanges channels away from the identified content, the fingerprints willstop matching, and when the user changes channels back to the identifiedcontent, the fingerprints will resume matching.

In operation 950, in response to the determination that the secondchange is a return to the first video data, the video overlay module 330resumes the use of the mask to cause display of a portion of the firstvideo data and a portion of the replacement video. Accordingly, the userof the client device 150 will perceive the return to the replacementvideo presented in operation 850.

FIG. 10 is a flowchart illustrating a method 1000, in some exampleembodiments, for dynamically providing a video overlay. The method 1000includes operations 810, 820, 830, 840, 1010, and 1020. The operations810-840 are described above, with respect to FIG. 8. By way of exampleand not limitation, the operations 1010 and 1020 are described as beingperformed by the components 110-190 of FIG. 1, the modules of 210-250 ofFIG. 2, and the modules 310-350 of FIG. 3.

In operation 1010, the video source identifier module 320 determines arate of playback of the identified video data. For example, the videoidentification system 190 may provide a set of expected fingerprints forthe video data that are compared to the fingerprints generated for thereceived video data. Accordingly, when the fingerprints for successiveframes of received video data match non-sequential expected fingerprints(e.g., every second or fourth fingerprint), a determination is made thatthe received video is playing at a different rate than (e.g., twice orfour times as fast as) the reference video used by the videoidentification system 190.

In operation 1020, the video overlay module 330 causes display of aportion of the video data received from the set-top box 140 and aportion of the replacement video based on the mask and the rate ofplayback. For example, as in operation 850, the mask is used todetermine which portion of the video received from the set-top box 140is passed through and which portion is replaced by the replacementvideo. Additionally, the rate of playback is used to control the rate ofplayback of the replacement video. In this manner, the user of theclient device 150 and the set-top box 140 perceives that playbackcontrols such as fast-forward, rewind, and pause operate on thereplacement video in the same manner as they would operate on theoriginally displayed video.

FIG. 11 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay. The method 1100includes operations 1110-1150. By way of example and not limitation, theoperations 1110-1150 are described as being performed by the components110-190 of FIG. 1, the modules of 210-250 of FIG. 2, and the modules310-350 of FIG. 3.

In operation 1110, the set-top box identifier module 310 of the clientdevice 150 accesses a video input stream containing content from acontent provider (e.g., the content source 110) and an intermediatedevice (e.g., the set-top box 140). For example, the set-top box 140 mayhave received a video stream from the content source 110 via a cableconnection, modified the video stream, and transmitted the modifiedvideo stream to the client device 150 via an HDMI connection. The videoinput stream may be accessed by the client device 150 as it is receivedor from data storage (e.g., from a solid-state drive integrated into theclient device 150).

The set-top box identifier module 310, in operation 1120, accesses atemplate corresponding to a screen portion generated by the intermediatedevice. For example, certain image manipulations performed by theintermediate device may be known to the client device 150. The imagemanipulations may correspond to particular portions of the screen. Toillustrate, the set-top box 140 may modify the video stream receivedfrom the content source 110 by placing a menu at the top or bottom ofthe screen. The template accessed by the set-top box identifier module310 corresponds to the screen portion containing video generated by theintermediate device.

In operation 1130, the set-top box identifier module 310 analyzes thevideo input stream to determine that a frame of the video input streammatches the template. For example, the template may indicate certainpixels that will be particular colors when the video input streamcontains the menu generated by the set-top box 140. Each incoming frameof video may be compared to the template to determine if the screenportion likely contains the menu. Some frames of video may not beanalyzed. For example, every other frame or every fifth frame may becompared to the template, thereby reducing the amount of processor timespent on the comparisons. In some example embodiments, operation 1130 isonly performed while replacement content is being presented, therebyreducing the opportunity for false positives to be generated.

The template matching may be performed using a pixel-by-pixel comparisonof a downsampled version of a frame of the video input stream with adownsampled version of the template. A pixel-distance between the(downsampled) frame and the template may be calculated. For example, thenumber of non-matching pixels may be used as the pixel difference.Alternatively, a color distance for each pixel may be calculated (e.g.,in the RGB; luminance and chroma (YUV); hue, saturation, and lightness(HSL); or hue, saturation, and value (HSV) color spaces) and the sum ofthe absolute values of the color distances taken as the pixel-distance.The determined pixel-distance is compared to a threshold to determine ifthe template is a match for the video input stream. In some exampleembodiments, the template and the video input stream arecross-correlated. Cross-correlation allows for the handling ofmulti-frame templates by determining a measure of similarity of theframe sequences when one sequence lags behind the other.

In some example embodiments, multiple templates are compared to thevideo input stream. For example, table 1640 of FIG. 16 may storetemplates for each of a plurality of set-top boxes. Each template in thetable 1640 includes a bitmap that indicates which pixels of the videoinput stream are to be compared with the pixels of the image of thetemplate. Even when the set-top box 140 has already been identified,multiple templates of multiple types are available to be compared to thevideo input stream. In some example embodiments, the templates aresorted by computational complexity, such that less computationallyexpensive templates are tested first. The first matching template may beused, halting the comparison process. Alternatively, all templates maybe compared and the best match (e.g., the template with the lowestpixel-distance from the video input stream frame) used.

The video overlay module 330, in operation 1140, generates a videooutput stream based on the determination that the frame of the videoinput stream matches the template. For example, if the content from thecontent source 110 is being replaced by the client device 150 withreplacement content and a determination is made that the set-top box 140is providing supplemental content, such as a menu, the video outputstream may be modified to include the menu when the menu is detected,such as by applying a suitable overlay mask to allow the menu to appearin the video output stream.

In operation 1150, the video overlay module 330 causes the video outputstream to be presented on a display (e.g., a display of the clientdevice 150). Some client devices 150 provide an application programminginterface (API) that allows for the video overlay module 330 to specifya portion of the display in which to present the video input stream andanother portion of the display in which to present replacement videocontent. For example, the portions may be specified using the cornercoordinates of a rectangular region, by using a 1-bit transparency maskthat indicates on a pixel-by-pixel basis whether the video input streampixel or the replacement pixel should be used, or by using an alpha maskthat indicates on a pixel-by-pixel basis a degree of blending betweenthe video input stream pixel and the replacement pixel. For those clientdevices 150, the API may be used. Otherwise, any partial blending orreplacement is performed by the video overlay module 330.

FIG. 12 is a flowchart illustrating a method 1200, in some exampleembodiments, for dynamically providing a video overlay. The method 1200includes operations 1210-1240. By way of example and not limitation, theoperations 1210-1240 are described as being performed by the modules of210-250 of FIG. 2 and the modules 310-350 of FIG. 3.

In operation 1210, the set-top box identifier module 310 of the clientdevice 150 accesses a video input stream that includes first content anda semi-transparent overlay overlaid at least partially on the firstcontent in an overlay zone. For example, the first content may becontent from a content provider (e.g., the content source 110) and thesemi-transparent overlay may be provided by an intermediate device(e.g., the set-top box 140). For example, the set-top box 140 may havereceived a video stream from the content source 110 via a cableconnection, modified the video stream, and transmitted the modifiedvideo stream to the client device 150 via an HDMI connection. The videostream may be accessed by the client device 150 as it is received orfrom data storage (e.g., from a solid-state drive integrated into theclient device 150).

The set-top box identifier module 310, in operation 1220, accesses atemplate that includes an indication of the overlay zone in the videoinput stream. For example, certain image manipulations performed by theintermediate device may be known to the client device 150. The imagemanipulations may correspond to particular portions of the screen. Toillustrate, the set-top box 140 may modify the video stream receivedfrom the content source 110 by placing a semi-transparent menu at thetop or bottom of the screen. The template accessed by the set-top boxidentifier module 310 corresponds to the screen portion containing thesemi-transparent overlay generated by the intermediate device (e.g., theset-top box 140).

The video source identifier module 320 may identify the content usingfingerprinting. For example, the received video data may include abroadcast TV signal overlaid with a semi-transparent menu generated bythe set-top box 140. Using the patch selection module 210 and the valuecalculation module 220 to generate one or more fingerprints for thereceived video data, the client device 150 sends the fingerprints to thevideo identification system 190. In response, the client device 150receives an identifier corresponding to the broadcast TV signal, whichserves to identify the portion of the received video data containing thebroadcast TV signal.

The video overlay module 330, in operation 1230, generates a videooutput stream that includes a modified semi-transparent overlay overlaidon replacement content. For example, if the content from the contentsource 110 is being replaced by the client device 150 with replacementcontent and a determination is made that the set-top box 140 has added asemi-transparent overlay to the content from the content source 110, thevideo output stream includes the replacement content and a modifiedversion of the semi-transparent overlay. In operation 1240, the videooverlay module 330 causes the video output stream to be presented on adisplay (e.g., a display of the client device 150).

The pseudocode methods below are representative of some alternativeembodiments:

semitransparentOverlayTransfer(inputVideoStream, identifiedVideoContent,overlayRegion) {  // inputVideoStream is the received video stream  //identifiedVideoContent is the content as transmitted from the content // provider, without the modifications performed by the intermediatedevice  // overlayRegion identifies the region defined by the templatethat was detected  // prior to invocation of this method,outputVideoStream contains  // replacement content, without thesemi-transparent overlay  for (each pixel in overlayRegion) {   //transfer the modifications performed by the intermediate device   // tothe replacement content   delta = inputVideoStream(pixel) −identifiedVideoContent(pixel);   outputVideoStream(pixel) += delta;  } }semitransparentOverlayTransfer(inputVideoStream, identifiedVideoContent,overlayRegion) {  // inputVideoStream is the received video stream  //identifiedVideoContent is the content as transmitted from the content // provider, without the modifications performed by the intermediatedevice  // overlayRegion identifies the region defined by the templatethat was detected  // prior to invocation of this method,outputVideoStream contains  // replacement content, without thesemi-transparent overlay  for (each pixel in overlayRegion) {   // ifthe pixel has been substantially changed by the intermediate   //device, use the changed pixel instead of the corresponding pixel of   //replacement content   delta = inputVideoStream(pixel) −identifiedVideoContent(pixel);   if (delta > PIXEL_OVERLAY_THRESHOLD)   outputVideoStream(pixel) = inputVideoStream(pixel);   } }semitransparentOverlayTransfer(inputVideoStream, identifiedVideoContent,overlayRegion, transparency) {  // inputVideoStream is the receivedvideo stream  // identifiedVideoContent is the content as transmittedfrom the content  // provider, without the modifications performed bythe intermediate device  // overlayRegion identifies the region definedby the template that was detected  // transparency is a value between 0and 1, indicating the degree of  // transparency of the semitransparentoverlay (0 indicating full opacity and  // 1 full transparency) as usedby the intermediate device  // prior to invocation of this method,outputVideoStream contains  // replacement content, without thesemi-transparent overlay  for (each pixel in overlayRegion) {   // theresult from the intermediate device is contentVideoStream *   //(transparency) + overlayPixel * (1 − transparency)   // solving foroverlayPixel gives:   overlayPixel = (inputVideoStream(pixel) −identifiedVideoContent(pixel) * transparency) / (1 − transparency);   //now apply the original overlay to our output   outputVideoStream(pixel)*= transparency;   outputVideoStream(pixel) += overlayPixel * (1 −transparency)  } } semitransparentOverlayTransfer(inputVideoStream,overlayRegion, transparency) {  // inputVideoStream is the receivedvideo stream  // overlayRegion identifies the region defined by thetemplate that was detected  // transparency is a value between 0 and 1,indicating the degree of  // transparency of the semitransparent overlay(0 indicating full opacity and  // 1 full transparency) to use in theoutput. The transparency may be, but  // need not be, the same as thatused by the intermediate device  // prior to invocation of this method,outputVideoStream contains  // replacement content, without thesemi-transparent overlay  // identify the color of the semi-transparentoverlay as the average color of  // the overlay region (e.g., randomun-modified pixels should average to a  // neutral gray, such as RGB(128, 128, 128), pixels modified by a semi-  // transparent overlayshould average to the color of the overlay)  // Alternatively, theoverlayColor may be included as data for the template rather  // thancalculated from the inputVideoStream  overlayColor = average(pixels inoverlay region);  // pixels in the overlay region are either backgroundpixels or foreground pixels.  // background pixels are those that havecolor values within a predetermined  // range of the overlay color;foreground pixels are those that are outside  // of the predeterminedrange  for (each pixel in overlayRegion) {   // the calculation of deltais represented as a simple subtraction, but   // could be implemented asa Euclidian distance in a color space   delta = inputVideoStream(pixel)− overlayColor;   // copy foreground pixels without change, but applytransparency to   // background pixels   if (delta >PIXEL_OVERLAY_THRESHOLD)    outputVideoStream(pixel) =inputVideoStream(pixel);   else {    outputVideoStream(pixel) *=transparency;    outputVideoStream(pixel) += overlayColor * (1 −transparency);   }  } } semitransparentOverlayTransfer(inputVideoStream,overlayRegion, transparency, textColor) {  // inputVideoStream is thereceived video stream  // overlayRegion identifies the region defined bythe template that was detected  // transparency is a value between 0 and1, indicating the degree of  // transparency of the semitransparentoverlay (0 indicating full opacity and  // 1 full transparency) to usein the output. The transparency may be, but  // need not be, the same asthat used by the intermediate device  // prior to invocation of thismethod, outputVideoStream contains  // replacement content, without thesemi-transparent overlay  // identify the color of the semi-transparentoverlay as the average color of  // the overlay region (e.g., randomun-modified pixels should average to a  // neutral gray, such as RGB(128, 128, 128), pixels modified by a semi-  // transparent overlayshould average to the color of the overlay)  // Alternatively, theoverlayColor may be included as data for the template rather  // thancalculated from the inputVideoStream  // textColor is a color value, thecolor value used for text by the intermediate  // device  overlayColor =average(pixels in overlay region);  for (each pixel in overlayRegion) {  // copy text without change, but apply transparency to backgroundpixels   // the pixels containing input text are, in the aggregate, thetext portion   // of the inputVideoStream   if (inputVideoStream(pixel)== textColor)    outputVideoStream(pixel) = inputVideoStream(pixel);  else {    outputVideoStream(pixel) *= transparency;   outputVideoStream(pixel) += overlayColor * (1 − transparency);   }  }} semitransparentOverlayTransfer(inputVideoStream, overlayRegion,transparency, messageType) {  // inputVideoStream is the received videostream  // overlayRegion identifies the region defined by the templatethat was detected  /// transparency is a value between 0 and 1,indicating the degree of  // transparency of the semitransparent overlay(0 indicating full opacity and  // 1 full transparency) to use in theoutput. The transparency may be, but  // need not be, the same as thatused by the intermediate device  // prior to invocation of this method,outputVideoStream contains  // replacement content, without thesemi-transparent overlay  // identify the color of the semi-transparentoverlay as the average color of  // the overlay region (e.g., randomun-modified pixels should average to a  // neutral gray, such as RGB(128, 128, 128), pixels modified by a semi-  // transparent overlayshould average to the color of the overlay)  // Alternatively, theoverlayColor may be included as data for the template rather  // thancalculated from the inputVideoStream  // messageType indicates aninformation type of text to be displayed  // E.g., MESSAGE_TIME =current time,  // MESSAGE_CHANNEL = current channel  overlayColor =average(pixels in overlay region);  for (each pixel in the overlayregion) {   // apply the overlay color to the region  outputVideoStream(pixel) *= transparency;   outputVideoStream(pixel)+= overlayColor * (1 − transparency);  }  // set message to a stringcontaining the appropriate information  message =getMessage(messageType);  // add the text of the message to the screen,for viewing by the user  printMessage(message, overlayRegion); }

FIG. 13 is a flowchart illustrating a method 1300, in some exampleembodiments, for dynamically providing a video overlay. The method 1300includes operations 1310-1360. By way of example and not limitation, theoperations 1310-1360 are described as being performed by the modules of210-250 of FIG. 2 and the modules 310-350 of FIG. 3.

In operation 1310, the client device 150 causes a video output stream tobe presented on a display. For example, the query fingerprint generator160 may have generated a query fingerprint from a video input streamreceived from the set-top box 140. Based on a generated queryfingerprint, replacement video content may have been received from thereplacement content server 180 and presented on a display of the clientdevice 150 as a video output stream.

The client device 150, in operation 1320, accesses a video input streamcontaining content from a content provider and an intermediate device.For example, the video input stream may include content from the contentsource 110 that has been modified to include additional content from theset-top box 140.

In operation 1330, the client device 150 accesses a templatecorresponding to a screen portion generated by the intermediate device.For example, the video overlay module 330 may access a record in thedatabase schema of FIG. 16 that contains data for the device identifierof the set-top box 140.

The video overlay module 330, in operation 1340, analyzes the videoinput stream to determine that at least a portion of content of a frameof the video input stream matches the template. In some exampleembodiments, multiple such templates are iterated over to determine ifany of the templates for the intermediate device correspond to thecontent of a frame of the video input stream received from theintermediate device. The template may correspond to an icon (e.g., aphone icon or a mail icon) added to the video input stream by theintermediate device, predetermined text (e.g., “phone” or “mail”) addedto the video input stream by the intermediate device, a window in aparticular location added to the video input stream by the intermediatedevice, or any suitable combination thereof.

The pseudocode method below is representative of some exampleembodiments:

findMatchingTemplate(inputVideoStream, deviceId) {  // inputVideoStreamis the received video stream  // device Id is the identifier of theintermediate device  matchingTemplate = NULL;  // matchingTemplate willstore the identified match, or NULL if none  while (matchingTemplate ==NULL && template=getNextTemplate(deviceId))  {   // iterate over all ofthe templates for the intermediate device   // until we have looked atthem all or we have found a match   // bitwise-AND the current frame ofthe inputVideoStream with the bitmap   // of the template. This willzero out the portions of the frame that are not used   // in determiningif this template matches   frame = inputVideoStream.currentFrame( ) &template.bitmap;   // determine the difference between the frame and thetemplate   // this may be the sum of the absolute values of the distancein RGB space for   // each compared pixel   delta = compareImage(frame,image);   // if the template matches the frame, this is the matchingtemplate   if (delta < TEMPLATE_MATCH_THRESHOLD)    matchingTemplate =template;  } }

The video overlay module 330, in operation 1350, modifies the videooutput stream to include the screen portion generated by theintermediate device, based on the match between the template and the atleast a portion of content of the frame of the video input stream. Forexample, if the set-top box 140 is adding a notification message to thecontent received from the content source 110, one or more frames of thevideo input stream will match a template for the notification. The videooverlay module 330 identifies the portion of the frames of the videoinput stream that contain the notification and copies the notificationto the video output stream. In some example embodiments, copying thenotification comprises copying each pixel of the video input stream inthe notification area, as identified by the template, to thecorresponding pixel of the video output stream. In other exampleembodiments, copying the notification comprises determining the contentsof the notification (e.g., through OCR or image recognition) andregenerating the contents in the video output stream. In this way, anotification from the set-top box 140 is presented to a viewer of theclient device 150 while the replacement content from the replacementcontent server 180 is being displayed. In operation 1360, the videooverlay module 330 causes the modified video output stream to bepresented on a display (e.g., a display of the client device 150).

In some example embodiments, modification of the video output streamceases when the video input stream no longer matches the template. Inother example embodiments, modification of the video output streamceases when the video input stream matches a second template associatedwith the end of the modification. For example, a notification from theset-top box 140 may have a signature frame in an animation of a closingof the notification that can be matched by a second template.Accordingly, when the second template is matched, the video outputstream is no longer modified to include the notification.

FIG. 14 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay. The method 1400includes operations 1310-1340, 1360, and 1450-1460. Operations 1310-1340and 1360 are described above, with respect to FIG. 13. By way of exampleand not limitation, the operations 1450-1460 are described as beingperformed by the modules of 210-250 of FIG. 2 and the modules 310-350 ofFIG. 3.

In operation 1450, the video overlay module 330 determines, based on thematch, that a speed of play of the video input stream has been modifiedfrom a default speed. For example, the template may indicate adouble-triangle pointing right in a particular portion of the videoinput stream. Based on this double-triangle being present (as determinedby the match between the video input stream and the template), the videooverlay module 330 determines that playback speed has been increased.Similarly, the template may indicate a double-triangle pointing left ora pause symbol, and the video overlay module 330 could determine thatplayback speed of the video input stream has been reversed or paused.

In some example embodiments, the template also indicates the rate ofplayback. For example, an “×2” or “×4” may be present, indicating thatthe playback speed is doubled or quadrupled. As another example, twotriangles may indicate double speed, three triangles may indicatequadruple speed, and so on. The particular meaning for a particulartemplate may depend on a set-top box 140 identified by the set-top boxidentifier module 310.

In other example embodiments, the template does not indicate the rate ofplayback. The rate of playback may be determined by comparingfingerprints of successive frames of the video input stream withfingerprints of successive frames of a reference stream. If thefingerprints of the successive frames of the video input stream matchevery other fingerprint of the reference stream, playback is occurringat double speed. If the fingerprints are in reverse order, playback isoccurring in reverse. If the fingerprints of the successive frames ofthe video input stream are doubled, playback is occurring at half speed.

The video overlay module 330 modifies a speed of play of the videooutput stream in operation 1460. In some example embodiments, the speedof play of the video output stream is modified to match the speed ofplay of the video input stream. Thus, if the speed of the video inputstream is doubled relative to the default speed, the speed of the videooutput stream is also doubled.

The various methods disclosed herein may be combined. For example,combining the method 1300 with the method 1400 results in a video outputstream that has a modified speed when the video input stream has amodified stream, wherein the video output stream also includes thescreen portion generated by the intermediate device (e.g., afast-forward indicator).

FIG. 15 is a flowchart illustrating a method, in some exampleembodiments, for dynamically providing a video overlay. The method 1500includes operations 1510-1590. By way of example and not limitation, theoperations 1510-1590 are described as being performed by the modules of210-250 of FIG. 2 and the modules 310-350 of FIG. 3.

In operation 1510, the client device 150 accesses a video input streamcontaining content from a content provider. For example, the video inputdevice 1714 (as shown in FIG. 17) of the client device 150 may beconnected to a video output of the set-top box 140. A video input streammay be sent from the set-top box 140 to the client device 150. The videoinput stream may include content from the content source 110 and contentgenerated by the set-top box 140. For example, the set-top box 140 mayprovide an informational overlay superimposed on part of a video streamreceived from the content source 110.

In operation 1520, the client device 150 selects a video output streambased on content of the video input stream. For example, the queryfingerprint generator 160 may generate a query fingerprint based on atleast a portion of one or more frames of the video input stream. Basedon the fingerprint, replacement video content is selected (e.g., througha database lookup, through communication with the replacement contentserver 180, or both) for use as a video output stream.

In operation 1530, the client device 150 causes the selected videooutput stream to be presented on a display. For example, the videooutput stream may be presented on a display of the client device 150.Alternatively, the video output stream may be sent by a video output ofthe client device 150 to a separate display device.

The video source identifier module 320, in operation 1540, determinesthat a change in the video input stream indicates a change in thecontent provider of the video input stream. For example, the queryfingerprint generator 160 may have access to a series of fingerprintsthat match the original content of the video input stream. When thefingerprints generated by the query fingerprint generator 160 cease tomatch the series of fingerprints for the original content, adetermination may be made that the content of the video input stream haschanged, which may indicate that the content provider of the video inputstream has also changed. For example, a user of the intermediate devicemay have changed channels or changed inputs (e.g., from a cable TVsource to a broadcast TV source). In either case, the content providerhas changed.

In response to the determination that the content provider of the videoinput stream has changed, the client device 150 causes the video inputstream to be presented on the display (operation 1550). Thus, thereplacement video output stream is presented as a replacement for theoriginal video input stream accessed in operation 1510, but not for thechanged video input stream detected in operation 1540.

In operation 1560, the client device 150 accesses a templatecorresponding to a screen portion generated by the intermediate device.For example, the video overlay module 330 may access a record in thedatabase schema of FIG. 16 that contains data for the device identifierof the set-top box 140.

The video overlay module 330, in operation 1570, analyzes the videoinput stream to determine that at least a portion of content of a frameof the video input stream matches the template. In some exampleembodiments, multiple such templates are iterated over to determine ifany of the templates for the intermediate device correspond to thecontent of a frame of the video input stream received from theintermediate device. The template may correspond to an icon (e.g., acable icon or a TV icon) added to the video input stream by theintermediate device, predetermined text (e.g., a number or alphanumericsequence indicating a channel) added to the video input stream by theintermediate device, a window in a particular location added to thevideo input stream by the intermediate device, or any suitablecombination thereof.

In operations 1580 and 1590, based on the match, the client device 150determines that a second change in the video input stream indicates areturn to the first content provider and, responsive to thedetermination, causes the presentation of the video output stream toresume. For example, if the original video input stream included contentfrom channel 15, which was identified as matching content to bereplaced, then replacement video content was presented in operation1530. After a change of channel, presentation of the replacement videocontent ceased in operation 1550. After detection of a return to theoriginal channel (e.g., by matching a template for an overlay providedby the intermediate device that indicates that the current channel is15), presentation of the replacement video content in the video outputstream resumes (operation 1590). Thus, by use of the method 1500,replacement content for one content provider can be provided whileallowing the user to view the content for other providers and withoutbeing disrupted by switching away from and returning to the contentprovider being replaced.

In some example embodiments, the video output stream is modified toinclude a portion of the video input stream that matches the template.For example, if a template of the topological form of FIG. 6 is matched,with a region 610 including channel information generated by the set-topbox 140, then the region 610 can be copied from the video input streamto the video output stream, allowing the information generated by theintermediate device to be presented while the video output stream ispresented.

FIG. 16 is a block diagram 1600 illustrating data structures, in exampleembodiments, for dynamically providing a video overlay. FIG. 16 shows aPictureInPicture table 1610, a MenuOverlay table 1620, a MenuBlend table1630, and a Template table 1640.

The PictureInPicture table 1610 includes fields for a device identifier(DeviceID), a template identifier (TemplateID), and window coordinates(WindowCoords). In some example embodiments, the PictureInPicture table1610 contains one row for each picture-in-picture template for eachset-top box. Thus, once a particular set-top box is identified by itsDeviceID and a particular template is identified by its TemplateID, thecoordinates for the position and size of the picture-in-picture displayof the replacement content can be retrieved. The PictureInPicture table1610 may be used to support templates topologically similar to thetemplate of FIG. 4.

The MenuOverlay table 1620 includes fields for a device identifier, atemplate identifier, and menu coordinates (MenuCoords). In some exampleembodiments, the MenuOverlay table 1620 contains one row for each menuoverlay template for each set-top box. Thus, once a particular set-topbox is identified by its DeviceID and a particular template isidentified by its TemplateID, the coordinates for the position and sizeof the portion of the video occupied by a menu overlay can be determinedfrom the MenuCoords. The MenuOverlay table 1620 may be used to supporttemplates topologically similar to the template of FIGS. 5 and 6.

The MenuBlend table 1630 includes fields for a device identifier, atemplate identifier, menu coordinates, and blend coordinates(BlendCoords). In some example embodiments, the MenuBlend table 1630contains one row for each menu blend template for each set-top box.Thus, once a particular set-top box is identified by its DeviceID and aparticular template is identified by its TemplateID, the coordinates forthe position and size of the portion of the video occupied by a menuoverlay can be determined from the MenuCoords, and the coordinates forthe position and size of the portion of the screen to be generated byblending the replacement video with the video from the set-top box canbe determined from the BlendCoords. The MenuBlend table 1630 may be usedto support templates topologically similar to the template of FIG. 7. Insome example embodiments, the MenuOverlay table 1620 is subsumed intothe MenuBlend table 1630, since when the BlendCoords define a region ofzero area, the MenuBlend table 1630 may be used to support templatestopologically similar to the templates of FIGS. 5 and 6.

The Template table 1640 includes fields for a device identifier, atemplate identifier, a type, a bitmap, and an image. The bitmap is thesame size as the image and a frame of the video input stream. Each valuein the bitmap indicates whether the corresponding pixel in the image andthe video input stream is to be used to determine if the templatematches the video input stream. The portions of the image correspondingto the “1” values of the bitmap are to be compared to the video inputstream. The portions of the image corresponding to the “0” values of thebitmap are black. The type indicates the type of the template, for useonce a match is found. For example, a type of PICTUREINPICTURE indicatesthat the template is for a picture-in-picture display, and that thewindow coordinates for the embedded picture may be retrieved from thePictureInPicture table 1610.

According to various example embodiments, one or more of themethodologies described herein may facilitate dynamic video overlays.Moreover, one or more of the methodologies described herein may reducecomputation time on a client device performing methods of providingdynamic video overlays, transmission time between a client device and aserver, memory storage requirements on a server performing methods ofdynamic video overlays, and computation time on a server performingmethods of dynamic video overlays.

When these effects are considered in aggregate, one or more of themethodologies described herein may obviate a need for certain efforts orresources that otherwise would be involved in dynamic video overlays.Efforts expended by a user in dynamic video overlays may be reduced byone or more of the methodologies described herein. Computing resourcesused by one or more machines, databases, or devices (e.g., within thenetwork environment 100) may similarly be reduced. Examples of suchcomputing resources include processor cycles, network traffic, memoryusage, data storage capacity, power consumption, and cooling capacity.

FIG. 17 is a block diagram illustrating components of a machine 1700,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium, acomputer-readable storage medium, or any suitable combination thereof)and perform any one or more of the methodologies discussed herein, inwhole or in part. Specifically, FIG. 17 shows a diagrammaticrepresentation of the machine 1700 in the example form of a computersystem and within which instructions 1724 (e.g., software, a program, anapplication, an applet, an app, or other executable code) for causingthe machine 1700 to perform any one or more of the methodologiesdiscussed herein may be executed, in whole or in part. In alternativeembodiments, the machine 1700 operates as a standalone device or may beconnected (e.g., networked) to other machines. In a networkeddeployment, the machine 1700 may operate in the capacity of a servermachine or a client machine in a server-client network environment, oras a peer machine in a distributed (e.g., peer-to-peer) networkenvironment. The machine 1700 may be a server computer, a clientcomputer, a PC, a tablet computer, a laptop computer, a netbook, aset-top box, a smart TV, a personal digital assistant (PDA), a cellulartelephone, a smartphone, a web appliance, a network router, a networkswitch, a network bridge, or any machine capable of executing theinstructions 1724, sequentially or otherwise, that specify actions to betaken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include acollection of machines that individually or jointly execute theinstructions 1724 to perform all or part of any one or more of themethodologies discussed herein.

The machine 1700 includes a processor 1702 (e.g., a central processingunit (CPU), a graphics processing unit (GPU), a digital signal processor(DSP), an ASIC, a radio-frequency integrated circuit (RFIC), or anysuitable combination thereof), a main memory 1704, and a static memory1706, which are configured to communicate with each other via a bus1708. The machine 1700 may further include a graphics display 1710(e.g., a plasma display panel (PDP), an LED display, an LCD, aprojector, or a CRT). The machine 1700 may also include an alphanumericinput device 1712 (e.g., a keyboard), a video input device 1714 (e.g., acomponent video input, a composite video input, or a high-definitionmultimedia interface (HDMI)), a storage unit 1716, a signal generationdevice 1718 (e.g., a speaker), and a network interface device 1720.

The storage unit 1716 includes a machine-readable medium 1722 on whichare stored the instructions 1724 embodying any one or more of themethodologies or functions described herein. The instructions 1724 mayalso reside, completely or at least partially, within the main memory1704, within the processor 1702 (e.g., within the processor's cachememory), or both, during execution thereof by the machine 1700.Accordingly, the main memory 1704 and the processor 1702 may beconsidered as machine-readable media. The instructions 1724 may betransmitted or received over a network 1726 (e.g., network 170 ofFIG. 1) via the network interface device 1720.

As used herein, the term “memory” refers to a machine-readable mediumable to store data temporarily or permanently and may be taken toinclude, but not be limited to, random-access memory (RAM), read-onlymemory (ROM), buffer memory, flash memory, and cache memory. While themachine-readable medium 1722 is shown, in an example embodiment, to be asingle medium, the term “machine-readable medium” should be taken toinclude a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storeinstructions. The term “machine-readable medium” shall also be taken toinclude any medium, or combination of multiple media, that is capable ofstoring instructions for execution by a machine (e.g., machine 1700),such that the instructions, when executed by one or more processors ofthe machine (e.g., processor 1702), cause the machine to perform any oneor more of the methodologies described herein. Accordingly, a“machine-readable medium” refers to a single storage apparatus ordevice, as well as “cloud-based” storage systems or storage networksthat include multiple storage apparatus or devices. The term“machine-readable medium” shall accordingly be taken to include, but notbe limited to, one or more data repositories in the form of asolid-state memory, an optical medium, a magnetic medium, or anysuitable combination thereof. “Non-transitory machine-readable media”refers to all machine-readable media except for transitory signals.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute hardwaremodules. A “hardware module” is a tangible unit capable of performingcertain operations and may be configured or arranged in a certainphysical manner. In various example embodiments, one or more computersystems (e.g., a standalone computer system, a client computer system,or a server computer system) or one or more hardware modules of acomputer system (e.g., a processor or a group of processors) may beconfigured by software (e.g., an application or application portion) asa hardware module that operates to perform certain operations asdescribed herein.

In some embodiments, a hardware module may be implemented mechanically,electronically, or any suitable combination thereof. For example, ahardware module may include dedicated circuitry or logic that ispermanently configured to perform certain operations. For example, ahardware module may be a special-purpose processor, such as an FPGA oran ASIC. A hardware module may also include programmable logic orcircuitry that is temporarily configured by software to perform certainoperations. For example, a hardware module may include softwareencompassed within a general-purpose processor or other programmableprocessor. It will be appreciated that the decision to implement ahardware module mechanically, in dedicated and permanently configuredcircuitry, or in temporarily configured circuitry (e.g., configured bysoftware) may be driven by cost and time considerations.

Accordingly, the phrase “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. As used herein,“hardware-implemented module” refers to a hardware module. Consideringembodiments in which hardware modules are temporarily configured (e.g.,programmed), each of the hardware modules need not be configured orinstantiated at any one instance in time. For example, where a hardwaremodule comprises a general-purpose processor configured by software tobecome a special-purpose processor, the general-purpose processor may beconfigured as respectively different special-purpose processors (e.g.,comprising different hardware modules) at different times. Software mayaccordingly configure a processor, for example, to constitute aparticular hardware module at one instance of time and to constitute adifferent hardware module at a different instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multiplehardware modules exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over appropriate circuits and buses)between or among two or more of the hardware modules. In embodiments inwhich multiple hardware modules are configured or instantiated atdifferent times, communications between such hardware modules may beachieved, for example, through the storage and retrieval of informationin memory structures to which the multiple hardware modules have access.For example, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions describedherein. As used herein, “processor-implemented module” refers to ahardware module implemented using one or more processors.

Similarly, the methods described herein may be at least partiallyprocessor-implemented, a processor being an example of hardware. Forexample, at least some of the operations of a method may be performed byone or more processors or processor-implemented modules. Moreover, theone or more processors may also operate to support performance of therelevant operations in a “cloud computing” environment or as a “softwareas a service” (SaaS). For example, at least some of the operations maybe performed by a group of computers (as examples of machines includingprocessors), with these operations being accessible via a network (e.g.,the Internet) and via one or more appropriate interfaces (e.g., an API).

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Some portions of the subject matter discussed herein may be presented interms of algorithms or symbolic representations of operations on datastored as bits or binary digital signals within a machine memory (e.g.,a computer memory). Such algorithms or symbolic representations areexamples of techniques used by those of ordinary skill in the dataprocessing arts to convey the substance of their work to others skilledin the art. As used herein, an “algorithm” is a self-consistent sequenceof operations or similar processing leading to a desired result. In thiscontext, algorithms and operations involve physical manipulation ofphysical quantities. Typically, but not necessarily, such quantities maytake the form of electrical, magnetic, or optical signals capable ofbeing stored, accessed, transferred, combined, compared, or otherwisemanipulated by a machine. It is convenient at times, principally forreasons of common usage, to refer to such signals using words such as“data,” “content,” “bits,” “values,” “elements,” “symbols,”“characters,” “terms,” “numbers,” “numerals,” or the like. These words,however, are merely convenient labels and are to be associated withappropriate physical quantities.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or any suitable combination thereof), registers, orother machine components that receive, store, transmit, or displayinformation. Furthermore, unless specifically stated otherwise, theterms “a” or “an” are herein used, as is common in patent documents, toinclude one or more than one instance. Finally, as used herein, theconjunction “or” refers to a non-exclusive “or,” unless specificallystated otherwise.

The invention claimed is:
 1. A system, comprising: a display; memorythat stores instructions and a plurality of templates; and one or moreprocessors of a client device configured by the instructions to performoperations comprising: accessing a video input stream that includesfirst video content from a content provider and second video contentgenerated by a set-top box device upstream of the one or moreprocessors, wherein, in a frame of the video input stream, the firstvideo content corresponds to a first screen portion of the display andthe second video content corresponds to a second screen portion of thedisplay; generating, from the first video content, a query fingerprint;accessing, based on a comparison of the query fingerprint and areference fingerprint, replacement video content provided by areplacement content source, wherein the replacement video contentdiffers from the first video content, wherein the replacement videocontent is accessed separately from the video input stream; identifying,based on the set-top box device, a template of the plurality oftemplates, wherein the identified template indicates that the secondscreen portion comprises an overlay for displaying the second videocontent generated by the set-top box device; comparing the video inputstream to the identified template to determine that the frame of thevideo input stream corresponds to the identified template; responsive toa determination that the frame of the video input stream corresponds tothe identified template, generating a video output stream comprising thesecond video content for the second screen portion and the videoreplacement content for the first screen portion to mimic a presentationof the first video content and the second video content in the videoinput stream; and causing the video output stream to be presented on thedisplay.
 2. The system of claim 1, wherein accessing the replacementvideo content comprises: identifying, based on the comparison, the firstvideo content provided by the content provider; and identifying thereplacement video content based on the identification of the first videocontent provided by the content provider.
 3. The system of claim 1,wherein accessing the replacement video content comprises requesting thereplacement video content from the replacement content source.
 4. Thesystem of claim 1, wherein the system is a smart television.
 5. Thesystem of claim 1, wherein generating the video output stream comprisescopying the second video content in the second screen portion from thevideo input stream to the video output stream on a pixel-by-pixel basis.6. The system of claim 1, wherein comparing the video input stream tothe identified template comprises: downsampling the frame of the videoinput stream to form a downsampled frame; determining across-correlation between the downsampled frame and the identifiedtemplate; performing a comparison of the cross-correlation to athreshold; and determining, based on the comparison, that the frame ofthe video input stream corresponds to the identified template.
 7. Thesystem of claim 1, wherein the operations further comprise: accessing anidentifier of the set-top box device; and selecting a subset of theplurality of templates based on the identifier of the set-top boxdevice, wherein comparing the video input stream to the identifiedtemplate to determine that the frame of the video input streamcorresponds to the identified template is a sub-step of comparing eachtemplate of the subset of the plurality of templates to determinewhether the frame of the video input stream corresponds to the template.8. A method comprising: accessing, at a display device, a video inputstream that includes first video content from a content provider andsecond video content generated by a set-top box device upstream of thedisplay device, wherein, in a frame of the video input stream, the firstvideo content corresponds to a first screen portion of the displaydevice and the second video content corresponds to a second screenportion of the display decice; generating, from the first video content,a query fingerprint; accessing, based on a comparison of the queryfingerprint and a reference fingerprint, replacement video contentprovided by a replacement content source, wherein the replacement videocontent differs from the first video content, wherein the replacementvideo content is accessed separately from the video input stream;identifying, based on the set-top box device, a template of theplurality of templates, wherein the identified template indicates thatthe second screen portion comprises an overlay for displaying the secondvideo content generated by the set-top box device; comparing the videoinput stream to the identified template to determine that the frame ofthe video input stream corresponds to the identified template;responsive to a determination that the frame of the video input streamcorresponds to the identified template, generating a video output streamcomprising the second video content for the second screen portion andthe replacement video content for the first screen portion to mimic apresentation of the first video content and the second video content inthe video input stream; and causing the video output stream to bepresented on the display device.
 9. The method of claim 8, whereinaccessing the replacement video content comprises: identifying, based onthe comparison, the first video content provided by the contentprovider; and identifying the replacement video content based on theidentification of the first video content provided by the contentprovider.
 10. The method of claim 8, wherein accessing the replacementvideo content comprises requesting the replacement video content fromthe replacement content source.
 11. The method of claim 8, wherein thedisplay device is a smart television.
 12. The method of claim 8, whereingenerating the video output stream comprises copying the second videocontent in the second screen portion from the video input stream to thevideo output stream on a pixel-by-pixel basis.
 13. The method of claim8, wherein comparing the video input stream to the identified templatecomprises: downsampling the frame of the video input stream to form adownsampled frame; determining a cross-correlation between thedownsampled frame and the identified template; performing a comparisonof the cross-correlation to a threshold; and determining, based on thecomparison, that the frame of the video input stream corresponds to theidentified template.
 14. The method of claim 8, further comprising:accessing an identifier of the set-top box device; and selecting asubset of the plurality of templates based on the identifier of theset-top box device, wherein comparing the video input stream to theidentified template to determine that the frame of the video inputstream corresponds to the identified template is a sub-step of comparingeach template of the subset of the plurality of templates to determinewhether the frame of the video input stream corresponds to the template.15. A non-transitory machine-readable medium that stores instructionswhich, when executed by one or more processors of a client device, causethe client device to perform operations comprising: accessing, by theone or more processors, a video input stream that includes first videocontent from a content provider and second video content generated by aset-top box device upstream of the one or more processors, wherein, in aframe of the video input stream, the first video content corresponds toa first screen portion of a display and the second video contentcorresponds to a second screen portion of the display; generating, fromthe first video content, a query fingerprint; accessing, based on acomparison of the query fingerprint and a reference fingerprint,replacement video content provided by a replacement content source,wherein the replacement video content differs from the first videocontent, wherein the replacement video content is accessed separatelyfrom the video input stream; identifying, based on the set-top boxdevice, a template of the plurality of templates, wherein the identifiedtemplate indicates that the second screen portion comprises an overlayfor displaying the second video content generated by the set-top boxdevice; comparing the video input stream to the identified template todetermine that the frame of the video input stream corresponds to theidentified template; responsive to a determination that the frame of thevideo input stream corresponds to the identified template, generating avideo output stream comprising the second video content for the secondscreen portion and the replacement video content for the first screenportion to mimic a presentation of the first video content and thesecond video content in the video input stream; and causing the videooutput stream to be presented on the display.
 16. The non-transitorymachine-readable medium of claim 15, wherein accessing the replacementvideo content comprises: identifying, based on the comparison, the firstvideo content provided by the content provider; and identifying thereplacement video content based on the identification of the first videocontent provided by the content provider.
 17. The non-transitorymachine-readable medium of claim 15, wherein accessing the replacementvideo content comprises requesting the replacement video content fromthe replacement content source.
 18. The non-transitory machine-readablemedium of claim 15, wherein generating the video output stream comprisescopying the second video content in the second screen portion from thevideo input stream to the video output stream on a pixel-by-pixel basis.19. The non-transitory machine-readable medium of claim 15, whereincomparing the video input stream to the identified template comprises:downsampling the frame of the video input stream to form a downsampledframe; determining a cross-correlation between the downsampled frame andthe identified template; performing a comparison of thecross-correlation to a threshold; and determining, based on thecomparison, that the frame of the video input stream corresponds to theidentified template.
 20. The non-transitory machine-readable medium ofclaim 15, wherein the operations further comprise: accessing anidentifier of the set-top box device; and selecting a subset of theplurality of templates based on the identifier of the set-top boxdevice, wherein comparing the video input stream to the identifiedtemplate to determine that the frame of the video input streamcorresponds to the identified template is a sub-step of comparing eachtemplate of the subset of the plurality of templates to determinewhether the frame of the video input stream corresponds to the template.